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Writing a faulty literature review is one of many ways to derail a dissertation. This article summarizes 
some pivotal information on how to write a high-quality dissertation literature review. It begins with a 
discussion of the purposes of a review, presents taxonomy of literature reviews, and then discusses the 
steps in conducting a quantitative or qualitative literature review. The article concludes with a 
discussion of common mistakes and a framework for the self-evaluation of a literature review. 


Writing a faulty literature review is one of many ways to 
derail a dissertation. If the literature review is flawed, the 
remainder of the dissertation may also be viewed as 
flawed, because “a researcher cannot perform significant 
research without first understanding the literature in the 
field” (Boote & Beile, 2005, p. 3). Experienced thesis 
examiners know this. In a study of the practices of 
Australian dissertation examiners, Mullins and Kiley 
(2002) found that, 


Examiners typically started reviewing a 
dissertation with the expectation that it would 
pass; but a poorly conceptualized or written 
literature review often indicated for them that 
the rest of the dissertation might have 
problems. On encountering an inadequate 
literature review, examiners would proceed to 
look at the methods of data collection, the 
analysis, and the conclusions more carefully. 
(Boote & Beile, 2005, p. 6) 


Given the importance of literature reviews in both 
dissertations and journal articles, it may be surprising 
that so many of them are faulty. Boote and Beile (2005) 
claim that “the dirty secret known by those who sit on 
dissertation committees is that most literature reviews 
are poorly conceptualized and written” (p. 4). Further, 
dissertations and theses are not the only types of 
publications that suffer from poor literature reviews. 
Many literature reviews in manuscripts submitted for 


publication in journals are also flawed—see Alton-Lee 
(1998), Grante and Graue (1999), and LeCompte, 
Klinger, Campbell, and Menck (2003). 


Given that so many literature reviews are poorly done, it 
is surprising there is not more published information on 
how to write a literature review. Boot and Beile (2005) 
write, 


Doctoral students seeking advice on how to 
improve their literature reviews will find little 
published guidance worth heeding. .. . Most 
graduate students receive little or no formal 
training in how to analyze and synthesize the 
research literature in their field, and they are 
unlikely to find it elsewhere. (p. 5) 


Not only is there a lack of published information to 
guide writers of literature reviews, the labor intensive 
process of writing one compounds the problem. Gall, 
Borg, and Gall (1996) estimate that completion of an 
acceptable dissertation literature review will take 
between three and six months of effort. 


The purpose of this guide is to collect and summarize 
the most televant information on how to write a 
dissertation literature review. I begin with a discussion of 
the purposes of a review, present Cooper’s (1988) 
Taxonomy of Literature Reviews, and discuss the steps 
in conducting a quantitative or qualitative literature 
review. A discussion of common mistakes and a 
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framework for the self-evaluation of literature reviews 
concludes the article. 


Purposes for Writing a Literature Review 


Conducting a literature review is a means of 
demonstrating an author’s knowledge about a particular 
field of study, including vocabulary, theories, key 
variables and phenomena, and its methods and history. 
Conducting a literature review also informs the student 
of the influential researchers and research groups in the 
field. Finally, with some modification, the literature 
review is a “legitimate and publishable scholarly 


document” (LeCompte & colleagues, 2003, p. 124). 


Apart from the above reasons for writing a review (Le., 
proof of knowledge, a publishable document, and the 
identification of a research family), the scientific reasons 
for conducting a literature review are many. Gall, Borg, 
and Gall (1996) argue that the literature review plays a 
role in: 


e delimiting the research problem, 
e seeking new lines of inquiry, 

e avoiding fruitless approaches, 

e gaining methodological insights, 


e identifying recommendations for further 
reseatch, and 


e seeking support for grounded theory. 


Hart (1998) contributes additional reasons for reviewing 
the literature, including: 


e distinguishing what has been done from what 
needs to be done, 


e discovering important variables relevant to the 
topic, 


e synthesizing and gaining a new perspective, 


e identifying relationships between ideas and 
practices, 


e establishing the context of the topic or problem, 
e rationalizing the significance of the problem, 

e enhancing and acquiring the subject vocabulary, 
e understanding the structure of the subject, 


e relating ideas and theory to applications, 
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e identifying the main methodologies and research 
techniques that have been used, and 


e placing the research in a historical context to 
show familiarity with state-of-the-art 
developments. (p. 27) 


Another purpose for writing a literature review not 
mentioned above is that it provides a framework for 
relating new findings to previous findings in the 
discussion section of a dissertation. Without establishing 
the state of the previous research, it is impossible to 
establish how the new research advances the previous 
reseatch. 


Taxonomy of Literature Reviews 


An effective method to begin planning a research review 
is to consider where the proposed review fits into 
Cooper’s (1988) Taxonomy of Literature Reviews. As 
shown in Table 1, Cooper suggests that literature 
reviews can be classified according to five 
characteristics: focus, goal, perspective, coverage, organization, 
and audience. In Table 1, each characteristic is listed on 
the left, with the levels of the characteristics on the right. 
In the paragraphs that follow, each of these literature 
review characteristics are described in more detail. 


Focus 


The first characteristic is the focus of the review. Cooper 
(1988) identifies four potential foci: research outcomes, 
research methods, theories, ot practices or applications. 


Literature reviews that focus on research outcomes are 
perhaps the most common. In fact, the Educational 
Resources Information Center (1982, p. 85) defines a 
literature review as an “information analysis and 
synthesis, focusing on findings and not simply bibliographic 
citations, summarizing the substance of the literature 
and drawing conclusions from it” (italics mine). The 
Educational Resources Information Center suggests 
that, in terms of a developing a research rationale, an 
outcomes-oriented review may help identify a lack of 
information on a particular research outcome, thus 
establishing a justifiable need for an outcome study. 


Methodological reviews concentrate on research 
methods—Cooper’s second focus category. In a 
methodological review, research methods in the chosen 
field are investigated to identify key variables, measures, 
and methods of analysis and inform outcomes-oriented 
research. The methodological review is also helpful to 
identify methodological strengths and weaknesses in a 
body of research, and examine how research practices 
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differ across groups, times, or settings. Methodological 
reviews, combined with outcome reviews, may also 
identify ways in which the methods inform the 
outcomes. A methodological review may also lead to 
sound rationale that can justify proposed dissertation 
reseatch, if it turns out that the previous research has 
been methodologically flawed. 


Table 1. Cooper’s Taxonomy of Literature Reviews 


Characteristic 


Research outcomes 
Reseatch methods 
Theories 
Practices or applications 
Integration 
(a) Generalization 
(b) Conflict resolution 
(c) Linguistic bridge-building 
Criticism 
Identification of central issues 


Neutral representation 
Espousal of position 
Exhaustive 

Exhaustive with selective citation 
Representative 

Central or pivotal 

Historical 

Conceptual 

Methodological 

Specialized scholars 

General scholars 
Practitioners or policymakers 
General public 


Perspective 


Coverage 


Organization 


Audience 


From “Organizing Knowledge Synthesis: A Taxonomy of 
Literature Reviews,” by H. M. Cooper, 1988, Knowledge in Society, 
1, p. 109. Copyright by Springer Science + Business Media. 
Reprinted with permission of Springer Science + Business 

edia. 


A review of theories, Cooper’s third focus, can help 
establish what theories already exist, the relationships 
between them, and to what degree the existing theories 
have been investigated. A theoretical review is 
appropriate if, for example, the dissertation aims to 
advance a new theory. In terms of the research rationale, 
a theoretical review can help establish a lack of theories 
ot reveal that the current theories ate insufficient, 
helping to justify that a new theory should be put forth. 
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Finally, literature reviews can be focused on practices or 
applications. For example, a review might concentrate 
on how a certain intervention has been applied or how a 
group of people tend to carry out a certain practice. In 
terms of a research rationale, this fourth type of review 
can help establish a practical need not currently being 
met. 


While a dissertation review typically has a primary focus, 
it may also be necessary to address all or some of the foci 
mentioned above. For example, a review with an 
outcomes-oriented focus would likely also deal with the 
methodological flaws that might affect an outcome. An 
outcomes-oriented review may also deal with theories 
related to the phenomenon being investigated and 
introduce the practical applications of the knowledge 
that will ultimately be gained from the dissertation. 


Goal 


The goal of many reviews is to integrate and generalize 
findings across units, treatments, outcomes, and settings; 
to resolve a debate within a field; or to bridge the 
language used across fields. Meta-analysis, for example, 
is an often-used review technique in which the primary 
goal is to integrate quantitative outcomes across studies. 
In other reviews the goal may be to critically analyze 
previous research, identify central issues, or explicate a 
line of argument within a field. 


A dissertation review often has multiple goals. If the 
dissertation is solely a review, the author may be 
primarily interested in integration, but it also may be 
necessaty to critically analyze the research, identify 
central issues, or explicate an argument. However, if a 
dissertation author is using the literature review to justify 
a later investigation, the goal will place more emphasis 
on critically analyzing the literature, perhaps to identify a 
weakness and propose to remedy that weakness with 
dissertation research. Either way, the author must 
integrate reviews to present the reader with the big 
picture. Without integration, the map of the research 
landscape would be as large as the research landscape 
itself. 


Perspective 


In qualitative primary research, review authors often 
decide to reveal their own preexisting biases and discuss 
how those biases might have affected the review. Or, as 
is often the case in quantitative primary research, authors 
can attempt to take a neutral perspective and present the 
review findings as fact. The perspective taken depends 
largely on whether the review is conducted in the 
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quantitative or qualitative traditions. Since secondary 
reseatch (i.e., review research) methods parallel primary 
reseatch methods, it makes sense for the author of a 
qualitative review to follow the qualitative tradition and 
reveal biases and the author of a quantitative review to 
follow the quantitative tradition and claim a neutral 
position. This decision will be dictated by the particular 
case. 


Coverage 


Deciding how wide to cast the net is a critical step in 
conducting a review. Cooper proposes four coverage 
scenarios. In an exhaustive review, the reviewer promises to 
locate and consider every available piece of research on a 
cettain topic, published or unpublished. However, 
finding every piece of research could take more time 
than is available. The key to the exhaustive review is to 
define the population in such a way that it is bounded 
and the number of articles to review is manageable. 
Cooper (1988) calls this an exhaustive review with selective 
citation. For example, the reviewer might choose only to 
look at articles published in journals, but not conference 
papers; however, a theoretical reason to exclude 
conference papers is advised. 


A third coverage approach is to consider a representative 
sample of articles and make inferences about the entire 
population of articles from that sample. However, 
random sampling is far from foolproof. A perhaps more 
certain approach is to gather evidence that demonstrates 
that the representative sample is actually representative. 
The most sound approach may be to do both. 


Cooper’s fourth article selection approach is to take a 
purposive sample 10 which the reviewer examines only the 
central or pivotal articles in a field. The key here is to 
convince the reader that the selected articles are, in fact, 
the central or pivotal articles in a field, and just as 
importantly that the articles not chosen are not central or 
pivotal. 


Organization 


There are many formats in which to organize a review. 
Three of the most common ate the bistorical format, the 
conceptual format, and the methodological format. In the 
historical format the review is organized chronologically. 
Clearly, this is preferred when the emphasis is on the 
progression of research methods or theories, or on a 
change in practices over time. 


A second common organizational scheme is built 
around concepts. For example, the review may be 
organized around the propositions in a research 
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rationale or, in a theoretically-focused review, organized 
according to the various theories in the literature. Finally, 
the literature review can be organized methodologically, 
as in an empirical paper (i.e., introduction, method, 
results, and discussion). In some cases, it may be most 
effective to mix and/or match these organizational 
formats. For example, the reviewer might begin with an 
introduction, define the method, and present the results 
in a historical or conceptual format, then move on to the 
discussion of results. This organizational format is often 
used in meta-analytic reports. 


Audience 


The final characteristic of Cooper’s (1988) Taxonomy of 
Literature Reviews is audience. For a dissertation, the 
supervisor and reviewers of the dissertation are the 
primary audience. The scholars within the field that the 
dissertation relates to are the secondary audience. Avoid 
writing the dissertation literature review for a general, 
non-academic audience. What constitutes a good book is 
probably not what constitutes a good dissertation, and 
vice versa. 


How to Conduct a Literature Review 


Take a look at the list below. Does it look familiar? It 
could be a step-by-step guide on how to conduct 
ptimary research, but in fact it describes the stages of 
conducting a literature review (see Cooper, 1984). 


1. Problem formulation 

2. Data collection 

3. Data evaluation 

4, Analysis and interpretation 
5. Public presentation 


If one thing must be realized about conducting and 
reporting a literature review it is that the stages for 
conducting and reporting a literature review 
parallel the process for conducting primary 
research. With a few modifications, what one knows 
about conducting primary tfesearch applies to 
conducting secondary research (i.e., a literature review). 
The key components are (a) a rationale for conducting 
the review; (b) research questions or hypotheses that 
guide the research; (c) an explicit plan for collecting data, 
including how units will be chosen; (d) an explicit plan 
for analyzing data; and (e) a plan for presenting data. 
Instead of human participants, for example, the units in 
a literature review are the articles that are reviewed. 
Validity and reliability, the same issues that apply to 
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—r_—e 2. The Research Stages in Conducting a Literature Review 


as e 
8 Problem formation, Data collection \ Data evaluation 
Characteristics 


What 
procedures 
should be used 
to find relevant 
evidence? 


What evidence 
should be 
included in the 
review? 


Research 
questions asked 


Primary function 
in review 


Constructing 
definitions that 
distinguish 
relevant from 
irrelevant 
studies. 


Determining 
which sources 
of potentially 
relevant sources 
to examine. 


Differences in 
the reseatch 
contained in 
soutces of 
information. 


1. Differences in 
included 
operational 
definitions. 

2. Differences in 
operational 
detail. 


1. Narrow 
concepts might 
make review 
conclusions less 
definitive and 
robust. 

2. Superficial 
operational 
detail might 
obscure 
interacting 
variables. 


Procedural 
differences that 
create variation in 
teview conclusion 


Sources of 
potential invalidity 
in review 
conclusions 


1. Accessed 
studies might be 
qualitatively 
different from 
the target 
population of 
studies. 

2. People 
sampled in 
accessible 
studies might be 
different from 
target 
population of 


people. 


‘Anal lysis and 
interpretation 


Public presentation 


What retrieved |What procedures | What information 
should be used to |should be included in the 
make inferences |review report? 

about the 

literature as a 

whole? 


should be 
included in the 
review? 


Synthesizing valid 
retrieved studies. 


Applying editorial 
ctiteria to separate 
important from 
unimportant 
information. 


Applying 
criteria to 
separate “valid” 
from “invalid” 
studies. 


Differences in the 
tules of inference. 


1. Differences 
in quality 
criteria. 

2. Differences 
in the influence 
of non-quality 
criteria. 


Differences in guidelines 
for editorial judgment. 


1. Rules for 
distinguishing 
patterns from 
noise might be 
inappropriate. 

2. Review-based 
evidence might be 
used to infer 
causality. 


1. Omission of review 
procedures might make 
conclusions 
irreproducible. 

2. Omission of review 
findings and study 
procedures might make 
conclusions obsolete. 


1. Nonequality 
factors might 
cause improper 
weighting of 
study 
formation. 

2. Omissions in 
study reports 
might make 
conclusions 
unreliable. 


From “Scientific Guidelines for Conducting Integrative Research Reviews,” Review of Education Research, 1984, 52, pg. 293. Copyright 
1984 by Sage Publications. Reprinted with permission of Sage Publications. 


primary research, also apply to secondary research. And, 
as in primary research, the stages may be iterative and 
not necessarily completed in the order presented above. 


Table 2, from Cooper (1984), is a framework to guide 
the completion of the four research stages of a literature 
review. On the left, the table identifies the general 
characteristics of each research stage: the research 
questions asked, the primary functions of each stage, the 
procedural differences that may lead to differing 
conclusions, and the potential sources of invalidity at 
each stage. For each of the characteristics, the remaining 


columns of the table pose key questions to guide the 
review writer in: problem formation, data collection, data 
evaluation, analysis and interpretation, and public presentation. 
Following sections discuss in more detail the steps 
Cooper (1984) suggests for conducting a literature 
review. 


Problem formulation (for the literature review) 


Once the appropriate type of review has been identified 
(see Cooper’s taxonomy in Table 1), the focus shifts to 
problem formulation. In this step the reviewer decides 
what questions the literature review will answer and 
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determines explicit criteria to dictate the inclusion, or 
exclusion, of an article included in the review. At this 
point it is important to make a distinction between 
literature review questions (i.e., questions that can be 
answered by reviewing the secondary research) and 
empirical research questions (i.e., questions that can be 
answered only through primary research). The literature 
review is the primary source of the empirical research 
question (Randolph, 2007c). 


Problem formation begins with the determination of the 
questions that will guide the literature review. These 
questions should be influenced significantly by the goal 
and focus of the review. For example, if the goal of the 
review is to integrate research outcomes, then a 
meaningful research question might be: From the previous 
literature, what is the effect of intervention X on outcomes Y and Zé 
If the goal is to critically analyze the research methods 
used in previous literature, questions might include: 
What research methods have been used in the past to investigate 
phenomenon X? and What are the methodological flaws of those 
methods? If the literature review focus is on theories and 
the goal is to identify central issues, then a legitimate 
research question might be: What are the central theories that 
have been used to explain phenomenon X? At this point it is 
wise to seatch for literature reviews that may have 
already answered these or related questions. 


The second step in problem formation is to explicitly 
determine the criteria for inclusion and exclusion. In other 
wotds, determine which articles will be included in the 
review and which articles will be excluded. The particular 
criteria are influenced by the review’s focus, goals, and 
coverage. Below is an example of the criteria for 
inclusion and exclusion used in a review of the research 


on the use of student response cards (Randolph, 2007b): 


Studies were included in the quantitative synthesis if 
they met each of the following criteria: 


1. The study reported means and standard 
deviations or provided enough information to 
calculate means and standard deviations for each 
condition. 


2. The use of write-on response cards, preprinted 
response cards, or both was the independent 
variable. 


3. Voluntary single-student oral responding (Le., 
hand raising) was used during the control 
condition. 


4. The study reported results on at least one of the 
following dependent variables: participation, 
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quiz achievement, test achievement, or intervals 
of behavioral disruptions. 


5. The report was written in English. 


6. The data from one study did not overlap data 
from another study. 


7. The studies used repeated-measures-type 
methodologies. 


8. For separate studies that used the same data (e.g., 
a dissertation and a journal article based on the 
same dataset), only the study with the most 
comprehensive reporting was included to avoid 
the overrepresentation of a particular set of data. 


(pp. 115-116) 


The inclusion/exclusion criteria should be explicit and 
comprehensive enough so that any article that comes to 
light could be included or excluded solely based on those 
criteria. Further, the criteria should include enough detail 
so that two people, given the same set of articles, would 
identify virtually the same subset of articles. In fact, in 
reviews where reliability is essential, such as when an 
entire dissertation or thesis is a review, researchers often 
recruit other individuals to test the reliability of the 
inclusion/exclusion system, then compare the resultant 
subsets to reveal inconsistencies, revising the criteria 
accordingly. 


It is likely that creating a valid set of inclusion/exclusion 
criteria will require considerable trial and error pilot 
testing. Often, ambiguities in the criteria will result in 
atticles that are inadequately omitted. Recursively 
pilot-testing the criteria is time-consuming, but much 
less so than starting over after much data have been 
painstakingly collected and analyzed. 


Data collection 


The goal of the data collection stage is to collect an 
exhaustive, semi-exhaustive, representative, or pivotal 
set of relevant articles. As in primary research, the 
researcher of secondary data must not only devise a 
systematic plan for data collection, he or she must 
accurately document how the data were collected. The 
reviewer is advised to describe the data collection 
procedure with such detail that, theoretically, other 
reviewers following the same procedures under the same 
conditions would find an identical set of articles. 


The data collection process often begins with an 
electronic seatch of academic databases and the Internet. 
(Because relevant databases vary within fields, I will not 
discuss them here.) When these searches ate conducted, 
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careful, accurate records must be kept of the date of each 
seatch, the databases searched, the key words and key 
wotd combinations used, and the number of records 
resulting from each search. 


In my experience, electronic searches lead to only about 
ten percent of the articles that will comprise an 
exhaustive review. There are several approaches to 
locate the remaining 90%. The most effective method 
may be to search the references of the articles that were 
retrieved, determine which of those seem relevant, find 
those, read their references, and repeat the process until 
a point of saturation is reached—a point where no new 
relevant articles come to light. 


When electronic and reference searching is exhausted, 
the reviewer is advised to share the list of references with 
colleagues and experts in the field to determine if they 
detect any missing articles. Sending a query to the main 
Listserv of experts in the relevant field, with a request 
that they identify missing articles, is often effective to 
yield additional references. It is also advisable to share 
the final list of potentially relevant articles with 
dissertation supervisors and reviewers, as they, too, may 
be aware of additional relevant literature. 


The data collection process can stop when the point of 
satutation is reached, and the teviewer has sufficient 
evidence to convince readers that everything that can 
reasonably be done to identify all relevant articles has 
been diligently undertaken. Of course, it is likely that 
new articles will come to light after the data collection 
period has concluded. However, unless the new article is 
critically important, I suggest leaving it out. Otherwise, 
the reviewer may have to open the floodgates and start 
anew the data collection process. 


Now the reviewer must devise a system to further cull 
the collected articles. For example, to separate the 
potentially relevant from the obviously irrelevant 
studies, the reviewer might read every word of every 
electronic record, just the abstract, just the title, or some 
combination. Whichever method is chosen, the reviewer 
is advised to accurately document the process 
undertaken. When the obviously irrelevant articles have 
been identified and discarded, the reviewer can begin to 
determine which of the remaining articles will be 
included in the literature review. Again, when reliability 
is critical, it is common for two or more other qualified 
individuals to determine which articles in the new subset 
meet the criteria for inclusion and exclusion to estimate 
and consider the level of interrater agreement. 
(Neuendorf [2002] provides a thorough discussion of 
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methods to quantify interrater agreement.) When the 
reviewer is satisfied that the final subset of relevant 
articles is complete, the data evaluation stage can begin. 


Data evaluation 


In the data evaluation stage the reviewer begins to 
extract and evaluate the information in the articles that 
met the inclusion criteria. To begin, the reviewer devises 
a system for extracting data from the articles. The type of 
data extracted is determined by the focus and goal of the 
review. For example, if the focus is research outcomes 
and the goal is integration, one will extract research 
outcomes data from each article and decide how to 
integrate those outcomes. As the data are evaluated, the 
reviewer is advised to document the types of data 
extracted and the process used. Because it requires 
extensive detail, this documentation is sometimes 
recorded using separate coding forms and a coding 
book, which are included as dissertation appendices. Or, 
the documentation may be included within the main 
body of the dissertation. 


Whether the procedures for extracting the data are 
recorded in a separate coding book or included within 
the body of the dissertation, the level of detail should be 
such that, actually or theoretically, a second person could 
attive at more or less the same results by following the 
recorded procedute. 


A coding book is an electronic document, such as a 
spreadsheet, or a physical form on which data are 
recorded for each article. The coding book documents 
the types of data that will be extracted from each article, 
the process used to do so, and the actual data. If the 
focus of the research is on outcomes, for example, the 
coding book should include one or more variables that 
track the extraction of research outcomes. The literature 
review, of course, will require the extraction of 
additional types of data, especially data that identify the 
factors that may influence research outcomes. For 
example, in experimental research the reviewer’s coding 
book will extract from each article the measurement 
instruments used; the independent, dependent, and 
mediating/moderating variables investigated; the data 
analysis procedures; the types of experimental controls; 
and other data. Of course, the influencing factors vary 
depending on the topic. 


Examining previous literature reviews, meta-analyses, or 
coding books is helpful to understand the scope and 
organization of a coding book. A freely-downloadable 
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example of a coding book and coding sheet used in a 
methodological review dissertation can be found from 
Randolph (2007a). 


It is essential to carefully consider the types of data to be 
extracted from each article, and to thoroughly pilot test 
the coding book. The extraction process tends to reveal 
other types of data that should be extracted, and may 
necessitate revision of the coding book and the recoding 
all articles. Further, if interrater reliability is important, 
the reviewer should alternately pilot test and revise the 
coding book until acceptable levels of interrater 
reliability are achieved. 


Literature reviews commonly examine data about the 
quality of research. However, there are conflicting views 
about the inclusion of low quality articles in a review. 
(See Table 3, at the end of this article, for a rubric on 
rating the quality of articles.) Some, like Cooper, suggest 
including only high quality articles in a study. Others 
suggest including both high quality and low quality 
studies and reporting the differences between the two. If 
there is not a difference, the data can be grouped 
together. If there is a difference, however, the reviewer 
may want to separately report results from the 
high-quality articles and low-quality articles. 


A goal of many reviews is to integrate or synthesize 
reseatch outcomes. Thus, a common metric or measure 
must be identified into which all of the research 
outcomes can be translated. In a quantitative synthesis, 
for example, the common metric might be the difference 
in proportions between control and treatment groups. 


Data analysis and interpretation 


Finally, at the data analysis and interpretation stage, the 
reviewer attempts to make sense of the extracted data. If 
the goal of the literature review is integration, the 
reviewer now integrates the data. Depending on the type 
of data extracted, a quantitative, qualitative, or 
mixed-methods synthesis will be performed. More 
information about analyzing data for quantitative and 
qualitative literature reviews is given later. 


Public presentation 


At this stage the review author determines which 
information is more important and will be presented and 
which information is less important information and can 
be left out. In a dissertation literature review, the author 
can be liberal about how much information to include. 
As discussed earlier, literature reviews are commonly 
organized historically, conceptually, or methodologically. 
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As mentioned earlier, the primary audience for the 
literature review is the dissertation supervisor and other 
dissertation reviewers. The secondary audience is other 
scholars in the field. The dissertation review can be 
revised later to meet the needs of a more general 
audience. 


Formulating and justifying empirical research 
questions 


The literature review, combined with the research 
problem, should lead to the formulation of empirical 
research questions. Although Cooper does not include 
this stage in his (1988) Taxonomy of Literature Reviews, 
it is an essential part of a dissertation. At this point, the 
dissertation author explains, using evidence from the 
review, how the dissertation makes a meaningful 
contribution to knowledge in the field. The American 
Education Research Association (2006) explains some 
of the ways new research can contribute to existing 
research: 


If the study is a contribution to an established 
line of theory and empirical research, it should 
make clear what the contributions are and how the 
study contributes to testing, elaborating, or 
enriching that theoretical perspective. 


If a study is intended to establish a new line of 
theory, it should make clear what that new theory 
is, how it relates to existing theories and evidence, 
why the new theory is needed, and the intended 
scope of its application. 


If the study is motivated by practical concerns, 
it should make clear what those concerns are, why 
they ate important, and how this investigation can 
address those concerns. 


If the study is motivated by a lack of 
information about a problem or issue, the 
problem formation should make clear what 
information is lacking, why it is important, and 
how this investigation will address the need for 
information. (p. 3) 


Quantitative Literature Reviews 


Two common types of quantitative reviews ate narrative 
reviews and meta-analytic reviews. Before the method of 
meta-analysis became prevalent, almost all quantitative 
reviews were narrative. According to Gall, Borg, and 
Gall (1996), narrative reviews: 


emphasized better-designed studies, and 
organized their results to form a composite 
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picture of the state of the knowledge on the 
problem or topic being reviewed. The number 
of statistically significant results, compared 
with the number of nonsignificant results, may 
have been noted. Each study may have been 
described separately in a few sentences or a 


paragraph. (pp. 154-155) 


However, despite their frequent use, narrative reviews 
tend to be significantly affected by the reviewer’s 
subjectivity. Research has indicated that the conclusions 
of one narrative review can differ completely from 
another review written by a different author, even when 
exactly the same articles are reviewed (Light & Pillemer, 
1984). 


Today, meta-analytic reviews have taken the forefront. 
In a meta-analytic review, the reviewer (a) collects a 
representative or comprehensive sample of articles, (b) 
codes those articles according to a number of aspects 
(e.g., study quality, type of intervention used, type of 
measute used, study outcomes), (c) finds a common 
metric (e.g., a standardized mean difference effect size) 
that allows the study outcomes to be synthesized, and 
then (d) examines how the characteristics of a study 
covaty with study outcomes. 


Figure 1, below, shows an example of a graph often used 
in meta-analysis. The forest plot illustrates the types of 
information typically yielded through meta-analyses. 
Figure 1, from Randolph 2007b, illustrates the outcomes 
of 13 studies that investigated the effects of response 
catds on academic achievement (in this case, quiz 
scores). The triangle represents the effect and the lines 
on either side indicate the 95% confidence intervals for 
that effect. The common metric used for the forest plot 
is a standardized mean difference effect size called 
Cohen’s d. At the bottom of the figure is the weighted 
average effect size (i.e., the integrated outcome) of all 13 
studies, approximately 1.1, which means that the 
students scored about 1.1 standard deviations higher on 
their quizzes when using response cards than when not 
using response cards. 


As illustrated in Figure 1, meta-analysis is a useful way to 
synthesize and analyze a body of quantitative research 
(Cooper & Hedges, 1994a; Glass, McGaw, & Smith, 
1981; Lipsey & Wilson, 2001; or Rosenthal, 1991, are all 
excellent guidebooks for conducting meta-analyses). 
However, criticisms of meta-analysis include that it is 
subject to publication bias (ie. that statistically 
significant results tend to be published more than 
nonstatistically significant results) and that is too 
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mechanistic. Some, such as Slavin (1986), wisely suggest 
combining meta-analytic and narrative techniques. For 
example, one might quantitatively synthesize each study, 
but also provide a thorough narrative description of 
particularly relevant studies. 


Christie, 2003 (5) 
Swanson, 1999 (6) 
Wheatley, 1986 (9) 

King, 1996 (12) 
Narayan, 1986 (20) 
Maheady, 2002 (21) 
Gardner, 1990 (22) 
Cavanaugh, 1992 (23) 
Marmelejo, 2004 (27) 
Al-Atrash, 1999(29) 
Reynolds, 2003 (32) 
Shabani, 2004 (52) 


c Upper Cl. 
= Rindfuss, 1997 (85) 
3 Lower C1. 
3S WEIGHTED AVERAGE 
fal y Effect size (d) 
2-1 it} 1 2 3 
Effect size (d) 


Figure 1. A forest plot of the effects of response cards 
on quiz achievement. From “Meta-Analysis of the 
Effects of Response Cards on Student Achievement, 
Participation, and Intervals of Off-Task Behavior,” by 
J. J. Randolph, 2007, Journal of Positive Behavior 
Interventions, 92), p. 121. Copyright 2007 by Sage 
Publications. Reprinted with permission of Sage 
Publications. 


Qualitative Literature Reviews 


When a body of literature is primarily qualitative, or 
contains a mixture of quantitative and qualitative results, 
it may be necessary to conduct a qualitative review, 
either alone or as a complement to a quantitative review. 
This section presents two methods for conducting 
qualitative literature reviews. The first method was first 
put forth by Ogawa and Malen (1991). The second 
method, which I put forth, borrows the method of 
phenomenological research and applies it to conducting 
a literature review. Another useful resource for 
conducting qualitative literature reviews, not described 
here, is Noblit and Hare (1988). 


Ogawa and Malen’s method 


Borg, Gall, and Borg (1996) have broken down Ogawa 
and Malen’s (1991) method into the eight steps 
discussed below. Note that these steps parallel the basic 
steps in qualitative research. 
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Step 1: Create an audit trail. In this step, the reviewer 
carefully documents all of the steps that are taken. The 
audit trail serves as documentation to make clear the 
evidence that supports each finding, where that evidence 
can be found, and how that evidence was interpreted. 


Step 2. Define the focus of the review. The problem formation 
stage mentioned earlier is similar to this step. In this 
stage the constructs of the review are defined and, 
thereby, it is determined what to include in the review 
and what to leave out. 


Step 3: Search for relevant literature. This step is similar to the 
data collection stage mentioned earlier. According to 
Ogawa and Malen (1991), in addition to qualitative 
reseatch reports, nonresearch reports such as memos, 
newspaper articles, or meeting minutes should also be 
included in the review and not necessarily regarded as 
having less value than qualitative research reports. 


Step 4: Classify the documents. In this step the reviewer 
classifies the documents according to the types of data 
they represent. For example, some documents might be 
first-hand reports of qualitative research, others may be 
policy statements about the issue in question, and still 
other types of data might describe projects surrounding 
the issue. 


Step 5: Create summary databases. This step is similar to the 
data evaluation stage. In this stage the reviewer develops 
coding schemes and attempts to reduce the information 
in the relevant documents. On this point, Borg, Gall, and 
Borg (1996) wrote, 


You cannot simply read all these 
documents, take casual notes, and then 
write a literature review. Instead, you will 
need to develop narrative summaries and 
coding schemes that take into account all 
the pertinent information in the 
documents. The process is iterative, 
meaning, for example, that you might need 
to develop a coding scheme, apply it to the 
documents, revise it based on this 
experience, and re-apply it. (p. 159) 


Step 6: Identify constructs and hypothesized causal linkages. After 
summaty databases have been created, the task is to 
identify the essential themes of the documents and 
create hypotheses about the relationships between the 
themes. The goal here, unlike meta-analysis, is to 
increase the understanding of the phenomena being 
investigated, not to integrate outcomes and identify 
factors that covary with outcomes. 
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Step 7: Search for contrary findings and rival interpretations. In 
the tradition of primary qualitative research, it is 
necessary to actively search for contrary findings and 
rival interpretations. One might, for example, reread the 
documents at this point to search for contrary evidence. 


Step 8: Use colleagues or informants to corroborate findings. The 
last step in Ogawa and Malen’s (1991) method, 
corroborating findings, also parallels primary qualitative 
research. In this step, one shares a draft of the report 
with colleagues and informants, such as the authors of 
the documents included in the review, requesting that 
they critically analyze the review. In this way, based on 
the extent of agreement among the informants, the 
reviewer can confirm the degree to which the review’s 
conclusions ate sound. 


The phenomenological method for conducting a 
qualitative literature review 


The goal of phenomenological research is to arrive at the 
essence of the lived experience of a phenomenon 
(Moustakas, 1994). Applied as a review technique, the 
goal is to arrive at the essence of researchers’ empirical 
experiences with a phenomenon. In __ first-hand 
phenomenology, the individuals who have experienced a 
cettain phenomenon are interviewed. In using 
phenomenology as a review technique, the unit of 
analysis is the research report rather than an individual 
who experienced the phenomenon. When using 
phenomenology as a review technique, the data come 
from an empirical research report rather than interview 
data. 


Not surprisingly, the steps of a phenomenological 
review mirror the steps of phenomenological research. 
Those steps are briefly described below: 


Step 1: Bracketing. In phenomenological research, the first 
step is to identify the phenomenon to be investigated. 
The researcher then “brackets” his or her experience 
with the phenomenon by explaining his or her own 
experiences with and positions on the phenomenon. 


Step 2: Collecting data. The next step is to collect data about 
the phenomenon. In _ primary phenomenological 
research, the researcher would interview a set of people 
who had experienced the phenomenon. In using the 
phenomenological method as a review tool, the reviewer 
would read the reports of scientists who have done 
research on the phenomenon. As in quantitative reviews, 
the reviewer still must decide on criteria for inclusion 
and define the research strategy. 
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Step 3: Identifying meaningful statements. The third step is to 
identify meaningful statements. The researcher might do 
this by highlighting empirical claims made about the 
phenomenon of interest and collecting those claims, 
word-for-word, in some kind of spreadsheet or 
qualitative software to make the data manageable. 


Step 4. Giving meaning. After identifying meaningful 
statements, the next step is to give meanings to those 
statements. That is, the reviewer might put the 
meaningful statements into categories and then interpret 
and paraphrase them as groups. 


Step 5. Thick, rich description. The final step is to create a 
thick, rich description of the essence of primary 
researchers’ experiences with the phenomenon. The goal 
is to describe the essence of the phenomenon as seen 
through the eyes of the researchers who investigated that 
phenomenon. 


Mistakes Commonly Made in Reviewing 
Research Literature 


In order to help the reviewer avoid mistakes in 
conducting a literature review, some of the most 
common mistakes are listed below. Gall, Borg, and Gall 
(1996) claim that the most frequent mistakes made in 
reviewing the literature are that the researcher: 


1. does not clearly relate the findings of the 
literature review to the researcher’s own study; 


2. does not take sufficient time to define the best 
descriptors and identify the best sources to use in 
review literature related to one’s topic; 


3. relies on secondary sources rather than on 
primary sources in reviewing the literature; 


4. unctritically accepts another researcher’s findings 
and interpretations as valid, rather than 
examining critically all aspects of the research 
design and analysis; 


5, does not report the search procedures that were 
used in the literature review; 


6. reports isolated statistical results rather than 
synthesizing them by chi-square or meta-analytic 
methods; and 


7. does not consider contrary findings and 
alternative interpretations in synthesizing 
quantitative literature. (pp. 161-162) 
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Evaluating a Literature Review 


Bootes and Beile (2005) have created a five-category 
rubric for evaluating a literature review. The categories 
are coverage, synthesis, methodology, significance, and rhetoric. 
The rubric is presented in Table 3, below. Boote and 
Beile used this scoring rubric to rate a random sample of 
30 education-related academic dissertations. Table 4 
shows a summary of their results. 


How does your literature review measure up? 
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Table 3. Boote and Beile’s Literature Review Scoring Rubric 


Category 


1. Coverage A. Justified criteria for 
inclusion and exclusion 


from review 


2. Synthesis B. Distinguished 
between what has been 
done in the field and 


what needs to be done 
C. Placed the topic or 


problem in the broader 
scholarly literature 


Criterion 


1 


Did not discuss the 
criteria for inclusion or 
exclusion 


Did not distinguish what|Discussed what has and 


has and has not been 
done before 


Topic not placed in 
broader scholarly 
literature 


D. Placed the research in|History of topic not 


the historical context of 


discussed 


Key vocabulary not 
discussed 


FP, Articulated important |Key variables and 


variables and 
phenomena relevant to 
the topic 


G. Synthesized and 


phenomena not 
discussed 


Accepted literature at 


gained a new perspective|face value 


on the literature 


3. Methodology H. Identified the main 
methodologies and 
research techniques that 
have been used in the 
field, and their 
advantages and 


disadvantages 


I. Related ideas and 
theories in the field to 
research methodologies. 


. Rationalized the 
practical significance of 
the research problem 


K. Rationalized the 
scholarly significance of 
the problem 


5. Rhetoric L. Was written with a 
coherent, clear structure 
that supported the 


review 


Reseatch methods not 
discussed 


Reseatch methods not 
discussed 


Practical significance of 
research not discussed 


2 


Discussed the literature 


included and excluded 


has not been done 


Some discussion of 
broader scholarly 
literature 


of topic 
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ustified inclusion and 
exclusion of literature 


Critically examined the 
state of the field 


Topic clearly situated in 
broader scholarly 


Key vocabulary defined |Discussed and resolved 


Reviewed relationships 


ambiguities in 
definitions 


Noted ambiguities in 


among key variables and |literature and proposed 


phenomena 


Some critique of 
literature 


Some discussion of 
research methods used 
to produce claims 


Some discussion of 
appropriateness of 
reseatch methods to 
warrant claims 


Practical significance 
discussed 


Scholarly significance of |Scholarly significance 


research not discussed 


Poorly conceptualized, 
haphazard 


discussed 


Some coherent structure 


new relationships 


Offered new perspective 


Critiqued research 
methods 


Critiqued 
appropriateness of 


appropriateness of 
research methods to 
warrant claims 
Critiqued scholarly 
significance of research 


Well developed, 
coherent 


From “Scholars before Researchers: On the Centrality of the Dissertation Literature Review in Research Preparation,” by D. N. Boote and 
P. Beile, 2005, Educational Researcher, 34(6), p. 8. Copyright 2005 by Sage Publications. Reprinted with permission of Sage Publications 
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Table 4. Results from Using the Literature Scoring 


Rubric on 30 Education-Related Dissertation Literature 
Reviews 


ustified criteria from inclusion and 
exclusion from teview 
Placed the research in the historical 2.33 
context of the field . 
Acquired and enhanced the subject 
2.33 
vocabulary 
Articulated important variables and 
: 2.33 
phenomena related to the topic 


Mean (SD) 


Synthesized and gained a new 
; 1.42 
perspective on the literature 
Identified the main methodologies 
and research techniques that have 


been used in the field, and their 
advantages and disadvantages 


Rationalized the scholarly 
significance of the research 
problem 


1.92 (0.79) 


This table was adapted from the text of “Scholars before 
Researchers: On the Centrality of the Dissertation Literature 
Review in Research Preparation,” by D. N. Boote and P. Beile, 
2005, Educational Researcher, 34(6), p. 13 
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